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Abstract 

We consider a class of optimal control problems of stochastic delay differential equations 
(SDDE) that arise in connection with optimal advertising under uncertainty for the intro- 
duction of a new product to the market, generalizing classical work of Nerlove and Arrow 
[sof. In particular, we deal with controlled SDDE where the delay enters both the state 
and the control. Following ideas of Vinter and Kwong [s^l (which however hold only in the 
deterministic case), we reformulate the problem as an infinite dimensional stochastic control 
problem to which we associate, through the dynamic programming principle, a second order 
Hamilton- Jacobi-Bellman equation. We show a verification theorem and we exhibit some 
simple cases where such equation admits an explicit smooth solution, allowing us to construct 
optimal feedback controls. 

1 Introduction 

In this paper we consider a class of stochastic optimal control problems where the state equation 
is a stochastic delay differential equations (SDDE). Such problems arise for instance in modeling 
optimal advertising under uncertainty for the introduction of a new product to the market, 
generalizing classical work of Nerlove and Arrow 30]. The main novelty in our model is that 
we deal also with delays in the control: this is interesting from the applied point of view and 
introduces new mathematical difficulties in the problem. 

Dynamic models in marketing have a long history, that begins at least with the seminal 
papers of Vidale and Wolfe [13] and Nerlove and Arrow Since then a considerable amount 
of work has been devoted to the extension of these models and to their application to problems 
of optimal advertising, both in the monopolistic and the competitive settings, mainly under 
deterministic assumptions. Models with uncertainty have received instead relatively less attention 
see Feichtinger, Hartl, and Sethi for a review of the existing work until 1994, Prasad and Sethi 
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31 1 for a Vidale and Wolf-like model in the competitive setting, and e.g. Grosset and Viscolani 



Marinelli [22| for recent work on the case of a monopolistic firm). Moreover, it has been 
advocated in the literature (see e.g. Hartl [i^], Feichtinger et al. 0| and references therein), as 
it reasonable to assume, that more realistic dynamic models of goodwill should allow for lags in 
the effect of advertisement. Namely, it is natural to assume that there will be a time lag between 
advertising expenditure and the corresponding effect on the goodwill level. More generally, a 
further lag structure has been considered, allowing a distribution of the forgetting time. 
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In this work we incorporate both lag structures mentioned above. We formulate a stochastic 
optimal control problem aimed at maximizing the goodwill level at a given time horizon T > 0, 
while minimizing the cumulative cost of advertising expenditure until T. 

This optimization problem is studied using techniques of stochastic optimal control in infinite 
dimension. In particular, we extend to the stochastic case a representation result, proved by 
Vinter and Kwong [34I in the deterministic setting, that allows to associate to a controlled SDDE 
with delays both in the state and in the control a stochastic differential equation (without delays) 
in a suitable Hilbert space. This in turn allows us to associate to the original control problem 
for the SDDE an equivalent (infinite dimensional) control problem for the "lifted" stochastic 
equation. 

We deal with the resulting infinite dimensional optimal control problem through the dynamic 
programming approach, i.e. through the study of the associated Hamilton-Jacobi-Bellman (HJB) 
equation. The HJB equation that arise in this case is an infinite dimensional second order 
semilinear PDE that does not seem to fall into the ones treated in the existing literature (see 
Section for details). Here we give some preliminary results for this equation. First of all we 
consider the particular case (but still interesting from the applied point of view) when the delay 
does not enter the control term. In this case the approach of Goldys and Gozzi and the 
forward-backward SDE approach of Fuhrman and Tessitore ^3]' E3l apply- We show how 
to apply the former in Section^. 

Moreover we consider the general case of delays in the state and in the control: since we do 
not know whether a regular solution exist, the natural approach would be the one of viscosity 
solutions. We leave the treatment of the viscosity solution approach to a subsequent work. 
Here we concentrate on the special case where regular solutions exist. In this case we prove a 
verification theorem and the existence of optimal feedbacks. Finally, we show through a simple 
example (but nevertheless still interesting in applications) that, possibly in special cases only, 
smooth solutions may exist, allowing to prove verification theorems and the existence of optimal 
feedback controls. 

Some further steps are still needed to build a satisfactory theory: concerning the viscosity 
solutions theory, it would be important to get an existence and uniqueness result and a verification 
theorem; concerning regular solutions, other cases where further regularity may arise should be 
studied. These issues are part of work in progress. 

As references on viscosity solutions of HJB equations in infinite dimensions, their connection 
with stochastic control and applications, we refer to e.g. Lions jl^, [i^], Swi§ch js^], Gozzi, 
Rouy, and Swi§ch Gozzi and Swi§ch [l^, Gozzi, Sritharan, and Switch 

Other approaches to optimal control problems for systems described by SDDEs without in- 
finite dimensional reformulation have been proposed in the literature: for instance, see Elsanosi 
01 and Larssen [3| for a more direct application of the dynamic programmin g pr inciple without 
appealing to infinite dimensional analysis, and Kolmanovskii and Shaikhet [23I for the linear- 
quadratic case. See also Elsanosi, 0ksendal, and Sulem (3] for some solvable control problems of 
optimal harvesting, and Elsanosi and Larssen 0| for an application in financial mathematics. 

The paper is organized as follows: in section |2l we formulate the optimal advertising problem 
as an optimal control problem for a SDDE with delays both in the state and the control. In 
section 121 we prove a representation result allowing us to "lift" this non-Markovian optimization 
problem to an infinite dimensional Markovian control problem. In section we study the simpler 
case of a controlled SDDE with delays in the state only, for which known results apply. Section 
121 deals with the general case of delays in the state and in the control, for which we only give 
the verification result. Finally, in sectional we construct a simple example of a controlled SDDE 
with delay in the state and in the control, whose corresponding HJB equation admits a smooth 
solution, hence there exists an optimal control in feedback form for the control problem. 
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2 The advertising model 



Our general reference model for the dynamics of the stock of advertising goodwill y{s), < s < T, 
of the product to be launched is given by the following controlled stochastic delay differential 
equation (SDDE), where z models the intensity of advertising spending: 



dy{s)= aoy{s)+ miOvis + + bozis) + bi{0z{s + 0dC 

J ~r J —r 

+adWo{s), 0<s<T 

[ y{0) = ri°; y{0 = viO, 40 = m e [-r, 0], 



ds 



(1) 



where the Brownian motion Wq is defined on a filtered probability space {0,,J^,¥ = {J^s)s>o,^), 
with F being the completion of the filtration generated by Wq , and z belongs to := L|([0, r];M+), 
the space of square integrable non-negative processes adapted to F. Moreover, we shall assume 
that the following conditions are verified: 

(i) ao < 0; 

(ii) ai(.)GL2([-r,0];M); 

(iii) bo > 0; 

(iv) eL2([-r,0];E+); 

(v) ?7° > 0; 

(vi) r/(-) > 0, with 7/(0) = 

(vii) 6{-) > 0. 

Here ao is a constant factor of image deterioration in absence of advertising, bo is a constant 
advertising effectiveness factor, ai(-) is the distribution of the forgetting time, and is the 
density function of the time lag between the advertising expenditure z and the corresponding 
effect on the goodwill level. Moreover, r]^ is the level of goodwill at the beginning of the advertising 
campaign, 7/(-) and 5{-) are the histories of the goodwill level and of the advertising expenditure, 
respectively, before time zero (one can assume r/(-) = 6{-) = 0, for instance). Note that when 
ai(-)! biii), and a are identically zero, equation reduces to the classical model of Nerlove and 
Arrow j30|. 

Finally, we define the objective functional 



J{z{.))=E 



My{T)) 



ho{z{s)) ds 



(2) 



where ipQ is a concave utility function with polynomial growth at infinity, /iq is a convex cost 
function which is superlinear at infinity i.e. 



lim 



h{z) 



*+oo z 



+00, 



and the dynamics of y is determined by (Q) . The problem we will deal with is the maximization 
of the objective functional J over all admissible controls in U. 

Remark 1 Note that in the general framework of delay equations the functions oi and bi are 
measures. Here we do not consider such general framework for two reasons: first taking ai and 6i 
to be L? already captures the applied idea of the problem; second, taking ai and hi to be measures 
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would introduce some technical difficulties that we do not want to treat here. More precisely this 
would create some problems in the infinite dimensional reformulation bringing unbounded terms 
into the state equation. Indeed, if bi = 0, the case where oi is a measure can be easily treated 
by a different standard reformulation. This fact allows us to treat the case of point delays in the 
state with no delays in the control. This will be the subject of section 4. 



3 Reformulation in Hilbert space 

Throughout the paper, X will be the Hilbert space defined as 

X = Rx L^([-r, 0];M), 



with inner product 



and norm 



.0 

{x,y)=xoyo+ xi{C)yiiO dC 

J —r 



\ 1/2 



xo\' + J ki(Oprfc) 



where xq and xi(-) denote the M-valued and the -L^([— r, 0]; M)-valued components, respectively, 
of the generic element x of X. 

In this sectfon we shah adapt the approach of Vinter and Kwong Q to the stochastic case to 
recast the SDDE ^ as an abstract stochastic differential equation on the Hilbert space X and 
so to reformulate the optimal control problem. 

We start by introducing an operator A : D(A) C X ^ X as follows: 

A: {xo,xi{^)) ^ (^aoxo + xi{0), ai{C)xo - '^^^^^^ ) a.e. ^ e [-r, 0]. 

The domain of A is given by 

D{A) = {(xo,xi(-)) G X : xi(-) G W^'\[-r, 0]-R), xi(-r) = O} . 
The operator A is the adjoint of the operator A* : D(A*) C X ^ X defined as 

A* : (xo,xi(-)) ^ {aoXo+ f ai{^)xi{^) d^,x[{-)j (3) 



on 



D{A*) = {(xo,xi(-)) G M X W^'\[-r, 0];R) : xq = xi(0)} . 



It is well known that A* is the infinitesimal generator of a strongly continuous semigroup (see, 
e.g., Chojnowska-Michalik j3| or Da Prato and Zabczyk 0]), therefore the same is true for A. 
We also need to define the bounded linear control operator B : U ^ X as 

B (^bou,bi{-)uj, (4) 

where U := M+. Sometimes we shall identify the operator B with the element (6o, bi{-)) G X. 

We introduce now an abstract stochastic differential equation on the Hilbert space X that is 
equivalent, in the sense made precise by Proposition IS to the SDDE (HJ): 

r dY{s) = {AY{s) + Bz{s)) ds + G dW^is) 

[ y(o) = x G X, 



4 



where G : M ^ X is defined by 

G : xq^ {axo,0), 

and z{-) G U. Using theorems 5.4 and 5.9 in Da Prato and Zabczyk 0|, we have that equation 
admits a unique weak solution (in the sense of 01, p. 119) with continuous paths given by 
the variation of constants formula: 

Y{s) = e'^x+ r e^'-^'>'^Bz{T)dT+ fe'^'-^^^G dWaij). 
Jo Jo 

In order to state equivalence results between the SDDE and the abstract evolution equation 
@, we need to introduce the operator M : X x L^([— r, 0];M) X defined as follows: 

M : {xo,xi{-),v(-)) ^ (xo,m(-)), 

where ^ ^ 

J —T J — r 

The following result is a generalization of Theorems 5.1 and 5.2 of Vinter and Kwong 

Proposition 2 LetY{) he the weak solution of the abstract evolution equation |3p with arbitrary 
initial datum x & X and control z{-) £ lA. Then, for t > r, one has, ¥-a.s., 

Y{t) = M{Yo{t),Yo{t + -),z{t + -)). 

Moreover, let {y{t), t > —r} be a continuous solution of the stochastic delay differential equation 
(PJj, and Y{-) be the weak solution of the abstract evolution equation ^ with initial condition 

x = M{r,',ri{-),6{-)). 

Then, for t > 0, one has, ¥-a.s., 

Y{t)=M{y{t),y{t + -),z{t + -)), 
hence y{t) = Yo{t), ¥-a.s., for all t > 0. 

Proof. Let x = {xo,xi) G D{A) (for general x the same result will follow by standard density 
arguments - see e.g. Vinter and Kwong Equation © can be written as 

dYoit) = (aoYoit) + Yiit){0) + boz{t)) dt + a dWo{t) 



d - - (6) 



dYi{tm = [ai{i)Yo{t) - ^Yiim) + hiOm) dt, 
[ Yo{0)=xo, Yi{0){-) = xi{-), 
therefore, P-a.s., 

Yo{t) = e*''»xo+ f e'^^-'^'"'Yi{s){0)ds+ f e'^^~'^''''boz{s) ds + T e(*~")"V dVFo(s) 
Jo Jo Jo 

Yi{tm = [mxim+ [ mt-s)a^{-)Yo{s)mds+ [ Mt-s)b,{-)z{s)mds, (?) 

Jo Jo 
where ^{t) is the semigroup of truncated right shifts defined as 

[mfi-)m-\ otherwise 
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for all f & L'^. Then for t > r can be rewritten, using the definition of $ and recalling that 
r) = 0, since x E D{A), as (P-a.s.) 

Yi{t){0 = ai{a)Yo{t + a-C)da+ bi{a)z{t + a - ^) da, (8) 

J —r J —r 

which is equivalent to 

Y{t) = M{Yo{t),Yo{t + -),z{t + -)), 

as claimed. 

Let us now prove the second claim: the L^-valued component of the weak solution of the 
evolution equation © with initial data 1^(0) = x = M{rp ,rj{-),5{-)). satisfies, for ^ — i G [— r, 0], 
?G [-r,0], t >0, 



ai{a)r}{t + a — ^) da + j hi{a)5{t + a — da, 



P-a.s., as follows from Q. We assume here r/(0) = rj^, without loss of generality (the general case 
follows by density arguments, as in Again by ((T)) and some calculations we obtain, P-a.s., 



Yiitm 



ai{a)Yo{t + a — ^) da + / bi{a)z{t + a — ^) da, 



where 



Mo 



viO, ee[-r,o], 



Observe that the definition of Y is well posed because of the assumption r/(0) = rj^, and because 
77" = Yq(0) by the definition of the operator M. In order to finish, we need to prove that Yq{-) 
satisfies the same integral equation (in the mild sense) as the solution y(-) to the SDDE i.e. 
that the following holds for all t > 0: 



Jo U~r J-r 



But this follows immediately from (jH)) with ^ = 0: 

J —r J —r 

which proves the claim. The fact that y{t) = Yq^I), P-a.s., for all t >0, easily follows 



ds. 



Using Proposition [21 we can now give a Hilbert space formulation of our problem. Since we 
want to use the dynamic programming approach, from now on we let the initial time vary, calling 
it t with < t < T. 

The state space is X = M x L^([— r, 0]; M), the control space is U = M+, the control strategy 
is z(-) G U. The state equation is (jSJ with initial condition at t, i.e. 



dY{s) = iAY{s) + Bz{s)) ds + G dW^i-s) 
Y{t) = xeX, 



(9) 



and its unique weak solution, given the initial data {t,x) and the control strategy z{-), will be 
denoted by Y{-;t, x, z{-)). The objective functional is 



J{t,x;z{-)) =E 



t,x 



T 

ip{Y{T,t,x;z{-)))+ / h{z{s))ds 



(10) 
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where the functions h : U ^M. and (p : X ^M. are defined as 

h{z) = -ho{z) 

The problem is to maximize the objective function J{t,y; z{-)) over all z{-) G U. We also define 
the value function V for this problem as 

V{t,x) = inf J{t,x;z{-)). 

Moreover, we shall say that z*{-) G W is an optimal control if it is such that 

V{t,x) = J{t,x;z*{-)). 

Following the dynamic programming approach we would like first to characterize the value func- 
tion as the unique solution (in a suitable sense) of the following Hamilton-Jacobi-Bellman equa- 
tion 

vt + - TriQv^^) + {Ax, v^) + Ho{v^) = 

z (11) 

v{T) = if, 



where Q = G*G and 



Ho{p) = sup{{Bz,p) + h{z)). 



Moreover we would like to find a sufficient condition for optimality given in terms of V (a so-called 
verification theorem) and a feedback formula for the optimal control z*. 

4 The case with no delay in the control 

In a model for the dynamics of goodwill with distributed forgetting factor, but without lags in 
the effect of advertising expenditure, i.e. with bi{-) = in (P), it is possible to apply both the 
approach of Hamilton-Jacobi-Bellman equations in spaces developed by Goldys and Gozzi , 
and the backward SDE approach of Fuhrman and Tessitore |lH . We follow here the first approach, 
showing that both the value function and the optimal advertising policy can be characterized 
in terms of the solution of a Hamilton-Jacobi-Bellman equation in infinite dimension. In fact 
we treat a somewhat different case assuming that the distribution of the forgetting factor is 
concentrated on a point. This can be treated by a different standard reformulation and with 
simpler computations and interpretations. In particular, we consider the case where the goodwill 
evolves according to the following equation: 

dy{s) = [aoy{s) + aiy{s - r) + boZo{s)] ds + a dWo{s), < s <T 

(12) 

y(0) = ??°; y(0 =^(0 vee [-r,0]. 

By the representation theorems for solutions of stochastic delay equations of Chojnowska-Michalik 
j^, one can associate to (|12j) an evolution equation on the Hilbert space X of the type 

dY{s) = iAY{s) + VQz{s)) ds + ^dW{s), 

(13) 

y(0) = y, 

where A : D{A) C X ^ X is defined as 

A : {xo,xi{-)) ^ (ao^o + aixi{-r),x[{-)) 
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on its domain 

D{A) = {{xo,xi{-)) e (M X W^'\[-r,0] 

moreover z = {a~^boZo, zi{-)) G M+ x L^([— r, 0]; 
defined as 

Q ■ {xo,xi{-)) 



with zi{-] 
(cr^xo,0); 



: xo = xi(0)} ; 

a fictitious control; Q : X 



X is 



W is an X- valued cylindrical Wiener process with W = {Wq,Wi), and Wi is a (fictitious) 
cylindrical Wiener process taking values in L'^([— r, 0]; M). Finally, y = {rp,ri{-)). 

Remark 3 Note that the operator A just introduced does not coincide with the A introduced 
in section |31 In fact, A here is exactly the A* defined there . Similarly, the initial datum of 
this section differs from that of section |21 Note also that the reformulation carried out in this 
section does not extend to the more general case of delay in the control, explaining why the more 
elaborate construction of the previous section is needed. 

We also note that the insertion of the fictitious control zi is not necessary here. We do it 
to keep the control space U equal to the state space X so the formulation falls into the results 
contained in Goldys and Gozzi However, it can be easily proved that the results of |0] still 
hold when the weaker condition B{U) C Q^i^X) is satisfied. 

The operator A is the infinitesimal generator of a strongly continuous semigroup {^(s), s > 0} 
(see again Chojnowska-Michalik More precisely, one has 

S{s){xo,xi{-)) = (y(s),y(s + 0l5ehr,o])> 
where ?/(•) is the solution of the deterministic delay equation 

dy{s] 



dt 

I y(o) 



= aoy{s) + aiy{s - r), 0<s<T 

xo; y{0 = MO v^g [-r,o]. 



(14) 



Moreover, the X-valued mild solution Y{-) = (yo(-), yi(-)) of (|13() is such that Yq{-) solves the 
original stochastic delay equation (|T2|) . 

As in section 131 we now consider an associated stochastic control problem letting the initial 
time t vary in [0, T]. The state equation is (|1.S|) with initial condition at t, i.e. 



dY{s) 
Y{t)- 

and the objective function is 



{AY{s) + ^/Qz{s)) ds + ./QdW{s) 



(15) 



X, 



J(t,x;zo(-)) =E 



t,x 



ho{zo{s))ds 



with y(-) obeying the SDDE (|12|) . Defining 

Hzo,zi{-)) = -ho{zo) 
^{xo,xi{-)) = ipo{xo), 

thanks to the above mentioned equivalence between the SDDE (|12|) and the abstract SDK (|ll-{j) . 
we are led to the equivalent problem of maximizing the objective function 



J{t,x;z{-)) = E 



t,x 



^{Y(T))+ / h{z{s))ds 



(16) 
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with Y subject to (|T3|) . 

Before doing so, however, following we need to assume conditions on the coefficients of H12() 
such that the uncontrolled version of l\l'S\i . i.e. 

dZ{t) = AZ{t) dt + ^/QdW{t), 

(17) 

Z{0) = X, 

admits an invariant measure. It is known (see Da Prato and Zabczyk Q) that 



ao < 1, ao < -ai < + ag, (18) 

where 7coth7 = ao, 7 G]0,7r[, ensures that H17() admits a unique non-degenerate invariant mea- 
sure ;U. 

Remark 4 The deterioration factor ao is always assumed to be negative, hence the first condition 
in H18() is not a real constraint. In general, however, it is not clear what sign ai should have. If 
ai is also negative, i.e. it can again be interpreted as a deterioration factor, condition (|18|) says 
that ai cannot be "much more negative" than ao- On the other hand, if ai is positive, then the 
second condition in (|18|) implies that the improvement effect as measured by ai cannot exceed 
the deterioration effect as measured by |ao|. 

Let us now define the Hamiltonian Hq : X — > M as 

Ho{p) = sup ({\/Qz,p)x + h{z)) 

and write the Hamilton- Jacobi-Bellman equation associated to the control problem (|16() : 

vt^\ Tr(Qi;^^) + {Ax, v^) + Ho{v^) = 

^ (19) 

u(T, x) = ip{x). 

If the Hamiltonian Hq is Lipschitz (which follows from the hypothesis on ho), if G L'^{X,fi) 
(which follows from the hypothesis on ip), and the operator A satisfies ()18p. then ()19() admits 
a unique mild solution v in the space L"^ {0,T;Wq^ {X , fi)) , as follows from Theorem 3.7 of [l^ . 
Moreover, Theorem 5.7 of ibid, guarantees that v coincides (^-a.e.) with the value function 



V(t, x) = inf E*'^' 



t,x 



V{Y{T))+ / h{z{s))ds 



(by z £ U we mean, with a slight abuse of notation, zq £ U), and that there exists a unique 
optimal control z*, i.e. 

V{t,x) = J{t,x;z*{-)) 

with 

z*{t) = DH0Qv{t,Y*m, (20) 
and Y* is the solution of the closed-loop equation 

dYis) = [AY{s) + y/QDH{bQv{s, Y{s))] ds + y/Q dW{t) 

(21) 

Y{t) = x. 

The gradient operator Dq is, roughly speaking, a "weakly closable" extension of the Malliavin 
derivative Q^I'^D acting on w]^{X,\i). For the exact definition and construction of Dq we refer 
the reader to [l^ . 
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Remark 5 The HJB equation (|19|) is "genuinely" infinite dimensional, i.e. it reduces to a finite 
dimensional one only in very special cases. For example, by the results of Larssen and Risebro 
[2^, ()19() reduces to a finite dimensional PDE if and only if oq = — ai. However, under this 
assumption, we cannot guarantee the existence of a non-degenerate invariant measure for the 
Ornstein-Uhlenbeck semigroup associated to (|17|) . Even more extreme would be the situation of 
distributed forgetting time: in this case the HJB is finite dimensional only if the term accounting 
for distributed forgetting vanishes altogether! 

5 Delays in the state and in the control 

We now consider the case when also delays in the control are present. The optimal control 
problem is the one described in section 2 with ai(-) 7^ 0, / 0. 

The Hamilton-Jacobi-Bellman equation associated to the problem is 



Unfortunately it is not possible, in general, to obtain regular solutions of the HJB equation 
()22() using the existing theory based on perturbations of Ornstein-Uhlenbeck semigroups (see e.g. 
Barbu and Da Prato 1], Da Prato and Zabczyk and Gozzi ^B])- The main problem is 
the lack of regularity properties of a suitable Ornstein-Uhlenbeck semigroup associated to the 
problem: in particular, the associated gradient operator is not closable and the semigroup is not 
strongly Feller (see Goldys and Gozzi [\3\ and Goldys, Gozzi, and van Neerven [l4l]'). 

As mentioned in section ^ if there is no lag in the effect of advertisement spending on the 
goodwill, i.e. if 6i(-) = 0, then both the ap proach of Goldys and Gozzi 13] and the backward 
SDE approach of Fuhrman and Tessitore [ll| can be applied, obtaining a characterization of the 
value function and of the optimal advertising policy in terms of the (unique) solution to (|22j) . 
Both approaches, however, fail in this more general case, since they require, roughly speaking, 
that the image of the control is included in the image of the noise, i.e. that B{U) C G(X), which 
is clearly not true. 

The only approach that seems to work in the general case of delays in the state and in the 
control is, to the best of our knowledge, the framework of viscosity solutions (see Lions |2^ . 
(2^ . [2^). However, while this approach gives a characterization of the value function in terms 
of the unique (viscosity) solution of the Hamilton-Jacobi-Bellman equation (|^^ . this solution is 
only guaranteed to be continuous, hence one can construct from it an optimal control only in a 
rather weak form, through the so called viscosity verification theorems (see Gozzi, Swiech, and 
Zhou [3). The study of this problem in the framework of viscosity solutions is the subject of a 
forthcoming publication. 

Here we want to prove a verification theorem in the case when regular solutions of the HJB 
equation are available. 

Definition 6 A function v is said to be 

• a classical solution of the HJB equation \2^) if v ^ C"^'^([0,T] x X) and v satisfies 




(22) 



where Q = G*G and 



Ho{p) = snv{{Bz,p) + h{z)). 



pointwise; 
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an integral solution if v S C°'^([0,T] x X), and moreover for t G [0,T] and x E D{A), one 
has 

cT 



ip{x) - v{t,x) + 



^Tr{Qvxx{s,x)) + {Ax,Vx{s,x)) + Ho{vx{s,x)) 



ds = (23) 



Theorem 7 (Verification Tlieorem) Let v be an integral solution of the HJB Iggj) and let V 

he the value function of the optimal control problem. Then 

(i) v>V on [0,r] X X. 

(a) If a control z €zU is such that, at starting point {t,x), 

Ho{vx{s,Y{s))) = snp{{Bz,Vx{s,Y{s))) + h{z)} = {Bz{s),Vx{s,Y{s))) + h{z{s)) 

zeu 

for almost every s G [t,T], P-a.e., then this control is optimal and v{t,x) = V{t,x). 

(Hi) If we know a priori that V = v, then (ii) is a necessary (and sufficient) condition of 
optimality. 

Although there is a standard way to prove such results, this version of the verification theorem 
is not contained in the existing literature. 

We give an idea of the method by sketching the proof in the case of bounded A. The case of 
unbounded A can be treated by approximating A with its Yosida approximations An, and then 
passing to the limit as n ^ +cxo (see e.g. Barbu and Da Prato 1]). 

Proof. Let A and B be bounded operators. The core of the job is to prove that, for every 
(t, x) G [0, T] X X and any z gU the following fundamental identity holds 

V {t, x) = J {t, X- z (•)) + r [Ho{vx{s, Y{s))) - {Bz{s),vx{s, Y{s))) - h{z{s))] ds, (24) 



where Y{s) := Y{s;t, x; z{-)). Once this is proved, the three claims of the theorem follow as 
straightforward consequences of the definitions of the Hamiltonian Hq, of value function and of 
optimal strategy. 

Let us then prove (^1]) . We first approximate f by a sequence of smooth function u„ that 
solve suitable approximating equations and that are such that 

Vfi > V, Vnx ^ Vx. 

This is possible e.g. using the same ideas of Goldys and Gozzi (l^ . Section 4. Then for a.e. 
s € ([t, T] n M), one may show that 

^Vn{s,Y{s)) = Vnt{s,Y{s)) + {y'{s),Vnx{s,Y{s))) (25) 
ds 

= Vnt{s,Y{s)) + {AY{s) + Bz{s),Vnx{s,Y{s))). (26) 

Since is a classical solution of a suitable approximating HJB equation (see again 13]), we have 

Vntis, Y{s)) + {AY{s),Vns{s, Y{s))) = -Ho{B*Vnx{s, Y{s))) - gn{s, Y{s)), 

where gn is a term appearing in the approximating HJB such that — > as n ^ cxd (see again 
0]). Substituting in and then adding and subtracting /i (s, z (s)), we obtain 

^Vn{s, Y{s)) = {z{s),B*Vnx{s, Y{s))) - Ho{B*Vnx{s, Y{s))) - <7n(s, Y{s)) 

ds 

= {z{s),B*Vnx{s,Y{s)))+h{z{s)) 

-Ho{B*Vnx{s, Y{s))) - gn{s, Y{s)) - h{z{s)). 
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Integrating between t and T we get 

Vn (T, Y (T)) - vn (t, x) + [gn {s, Y {s)) + h{z{s))] ds 

= Ms),B*{vn^{s,Y{s)))) + h{z{s)) - H^{B*Vnx{s,Y{s)))] ds, 

which gives the desired equahty (pij) after passing to the hmit as n ^ oo. ■ 
About feedback maps, the following theorem holds true. 

Theorem 8 Under the same hypotheses of theorem^ assume that there exists a measurable map 
r : ([0, T] n M) X X ^ U , such that T {t,p) is a maximum point of the map z i-^ {z, B*p) + h{z) 
for given {t,p) G ([0,T] H M) x X. Assume also that for every {t,x) there exists a (mild) solution 
Y* (•) of the closed loop equation 

dY*{T) = AY*{t) + Br{T,v^,{T,Y*{T))) + GdWit), r G [t,T]nR; 
Y*{t) = x, xeX, 

such that T 1-^ T{t,Vx{t,Y*(t))) belongs toU. Then the couple {Y*{-), z* (■)) is optimal whenever 
the control strategy satisfies the feedback relation 

z* {T)=r{r,v, {r,Y* (r))) . 

Proof. This is a straightforward consequence of (ii) in the Verification Theorem, as the control 
generated by the feedback relation satisfies (ii). ■ 



6 An example with explicit solution 



Here we restrict ourselves to some less general specification of the objective function, but still 
meaningful, for which the HJB equation admits a smooth solution, and therefore the value 
function and the optimal control can be completely characterized. 
Let us assume 

h{z) = -I3zl 

and 

tp[x) = 7^0, 

with /3, 7 > 0. Then we have 

Hcv{p, z) = {Bz,p) + h{z) = {B,p)z - (3z\ 

and 

{B,p) > 
{B,p) < 0, 

or equivalently, in more compact notation, 



Ho{p) = sup Hcv{z,p) = < 

zGf/ 



{B,p)' 
4/3 

0, 



Hoip) 



i{B,py 

4/3 
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We conjecture a solution of the HJB equation (|22|) of the form 

v{t, x) = {w{t),x) + c{t), te[0,T], xe X, 

where w^-) : [0,T] X and c(-) : [0, T] — > M are given functions whose properties we will study 
below. Hence for t G [0, T] and x G X we have, assuming that all objects are well defined, 

vt{t,x) = {w'{t),x) + c'{t) 

Vx{t,x) = W{t) 



and 



((B w(t))+)'^ 

{w'{t),x) + c'{t) + {Ax, w) + '^y^ ^ = 0, t G [0, T), x£X, 



(27) 



_ {w{T),x) +c{T) =7X0, x£X. 
It is clear that, if A*w{t) is well defined for all t G [0, T], (|27j) is equivalent to 

J x) + {A*w{t),x) =0, t G [0, T[ 

1 «;(r) = (7,0) 



and 



c'(t) + 



4/3 



0, tG[o,r[ 



(28) 



(29) 



I c(r) = 0. 
Since (|28j) must hold for all x, then it implies 

r u;'(t) + ^*u;(t) = 0, t G [0,r[ 
1 iz;(r) = (7,0). 

Recalling (jH)), we obtain that (PS)) is equivalent to 



and 



dt ' di 

^ wi{t,0) = wo{t), 
The solution of (j5T|) is given by 



.0 

w'oit) + aowoit) + ai{Owi{t,C)dC = 0, t e [0,T[ 

J —r 

wo{T) = 7 



\t,O + ^{t,O = 0, tG[0,rUG[-r,0[ 



(30) 



^G [-r,0[ 

t G [o,r]. 



(31) 



wi{t, = wo{t - 0l{t-g6[o,r]}, 



(32) 



from which one can solve equation (|3fl|) . obtaining wq{-). Note that, unfortunately, the function w 
is never in D(A*), except for the case when it is everywhere. However this does not exclude that 
the candidate v{t,x) = {w{t),x) + c{t) solves the HJB equation l\'22\i in some sense. Indeed it is 
an integral solution of (|22|) in the sense of definition Q, as it follows from the above calculations. 
Note that v £ C{[0,T]x X) and that it is twice differentiable in the x variable, i.e. it satisfies the 
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hypotheses of the verification theorem [3 Since a maximizer of the current- value Hamiltonian is 
given by z* = {B,p)~^ /{2f3), then it is immediately seen that the control 

which does not depend on Y*(t), is admissible, hence (ii) of the verification theorem [7| holds, and 
z*{-) is optimal. 
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